Method for Reward Modeling and Automatic Parking Policy Optimization via LLM-Based Semantic Assessment
HAN Cheng1,2, ZHUANG Hanyang3, LI Yueyuan1,2, WANG Chunxiang1,2, YANG Ming1,2
1. School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240; 2. Key Laboratory of System Control and Information Processing of Ministry of Education, Shanghai Jiao Tong University, Shanghai 200240; 3. Global College, Shanghai Jiao Tong University, Shanghai 200240
Abstract：Hybrid policy frameworks combining reinforcement learning with geometric planners achieve high success rates in automatic-parking tasks. However, oscillatory gear-shifting behavior is still observed in some complex scenarios, and behavioral preferences cannot be adequately represented by manually designed scalar reward functions. To address these problems, a method for reward modeling and automatic parking policy optimization with large language model-based semantic assessment is proposed as a four-stage policy-optimization framework. First, trajectory pairs containing oscillatory negative samples are mined from the hybrid policy system, and preference labels are generated by ensemble voting among four models. Then, a two-stage reward model is trained through environment-reward pre-training and preference-residual fine-tuning. Preferences of large language models for trajectory quality are converted into sortable reward scores. Finally, elite trajectories are selected by scenario for behavior cloning. Experiments on 500 simulation scenarios constructed from perception data of real parking lots show that the proposed method improves parking success rates and greatly reduces the average number of forward-reverse gear switches during each parking process.
[1] Reeds J A, Shepp L A.Optimal paths for a car that goes both forwards and backwards[J]. Pacific Journal of Mathematics, 1990, 145(2): 367-393. [2] Dolgov D, Thrun S, Montemerlo M, et al. Path planning for autonomous vehicles in unknown semi-structured environments[J]. International Journal of Robotics Research, 2010, 29(5): 485-501. [3] Lee S, Lim W, Sunwoo M. Robust parking path planning with error-adaptive sampling under perception uncertainty[J/OL]. Sensors, 2020, 20(12). https://doi.org/10.3390/s20123560. [4] Jiang M Y, Li Y Y, Zhang S A, et al. HOPE: a reinforcement learning-based hybrid policy path planner for diverse parking scena-rios[J]. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(5): 6130-6141. [5] Ma Y J, Liang W, Wang G Z, et al. Eureka: human-level reward design via coding large language models[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/2310.12931. [6] Liang J, Huang W L, Xia F, et al. Code as policies: language model programs for embodied control[C]//Proceedings of the IEEE International Conference on Robotics and Automation. Washington, USA: IEEE, 2023: 9493-9500. [7] Lee H, Phatale S, Mansoor H, et al. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback[C]//Proceedings of the 41st International Conference on Machine Lear-ning. San Diego, USA: JMLR, 2024: 26874-26901. [8] Dong H Z, Xiong W, Goyal D, et al. RAFT: reward ranked fine-tuning for generative foundation model alignment[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/2304.06767. [9] Gulcehre C, le Paine T, Srinivasan S, et al. Reinforced self-trai-ning(ReST) for language modeling[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/2308.08998. [10] Li B, Shao Z J.A unified motion planning method for parking an autonomous vehicle in the presence of irregularly placed obstacles[J]. Knowledge-Based Systems, 2015, 86: 11-20. [11] Zhang J R, Chen H, Song S Y, et al. Reinforcement learning-based motion planning for automatic parking system[J]. IEEE Access, 2020, 8: 154485-154501. [12] Tang X L, Yang Y Y, Liu T, et al. Path planning and tracking control for parking via soft actor-critic under non-ideal scenarios[J]. IEEE/CAA Journal of Automatica Sinica, 2024, 11(1): 181-195. [13] Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proceedings of the 35th International Conference on Machine Learning. San Diego, USA: JMLR, 2018: 1861-1870. [14] Han R H, Wang S, Wang S J, et al. NeuPAN: direct point robot navigation with end-to-end model-based learning[J]. IEEE Tran-sactions on Robotics, 2025, 41: 2804-2824. [15] Amodei D, Olah C, Steinhardt J, et al. Concrete problems in AI safety[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/1606.06565. [16] Ng A Y, Russell S.Algorithms for inverse reinforcement learning[EB/OL]. [2026-04-17]. https://ai.stanford.edu/~ang/papers/icml00-irl.pdf. [17] 彭浩楠,唐明环,查奇文,等.智能网联汽车基于逆强化学习的轨迹规划优化机制研究[J]. 北京理工大学学报, 2023, 43(8): 820-831. (Peng H N, Tang M H, Zha Q W, et al. Research on inverse reinforcement learning-based trajectory planning optimization mechanism for autonomous connected vehicles[J]. Transactions of Beijing Institute of Technology, 2023, 43(8): 820-831.) [18] Ng A Y, Harada D, Russell S. Policy invariance under reward transformations: theory and application to reward shaping[EB/OL]. [2026-04-17]. http://luthuli.cs.uiuc.edu/~daf/courses/Games/AIpapers/ml99-shaping.pdf. [19] Christiano P F, Leike J, Brown T B, et al.Deep reinforcement learning from human preferences[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 4302-4310. [20] Bradley R A, Terry M E.Rank analysis of incomplete block designs: I. the method of paired comparisons[J]. Biometrika, 1952, 39(3/4): 324-345. [21] Ouyang L, Wu J, Jiang X, et al.Training language models to fo-llow instructions with human feedback[C]//Proceedings of the 36th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2022: 27730-27744. [22] Lee K, Smith L M, Abbeel P.PEBBLE: feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training[C]//Proceedings of the 38th International Conference on Machine Learning. San Diego, USA: JMLR, 2021: 6152-6163. [23] Brown D S, Goo W, Niekum S.Better-than-demonstrator imitation learning via automatically-ranked demonstrations[C]//Proceedings of the Conference on Robot Learning. San Diego, USA: JMLR, 2020: 330-359. [24] 王祥,谭国真.基于知识与大语言模型的高速环境自动驾驶决策研究[J]. 系统仿真学报, 2025, 37(5): 1246-1255. (Wang X, Tan G Z.Research on decision-making of autonomous driving in highway environment based on knowledge and large language model[J]. Journal of System Simulation, 2025, 37(5): 1246-1255.) [25] Xie T B, Zhao S H, Wu C H, et al. Text2Reward: reward shaping with language models for reinforcement learning[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/2309.11489. [26] Kwon M, Xie S M, Bullard K, et al. Reward design with language models[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/2303.00001. [27] Wei J, Wang X Z, Schuurmans D, et al.Chain-of-thought promp-ting elicits reasoning in large language models[C]//Proceedings of the 36th International Conference on Neural Information Proce-ssing Systems. Cambridge, USA: MIT Press, 2022: 24824-24837. [28] Huang Z Y, Wu J D, Lü C.Efficient deep reinforcement learning with imitative expert priors for autonomous driving[J]. IEEE Tran-sactions on Neural Networks and Learning Systems, 2023, 34(10): 7391-7403. [29] Hu H Y, Mirchandani S, Sadigh D.Imitation bootstrapped reinforcement learning[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/2311.02198. [30] Fleiss J L.Measuring nominal scale agreement among many raters[J]. Psychological Bulletin, 1971, 76(5): 378-382. [31] Landis J R, Koch G G.The measurement of observer agreement for categorical data[J]. Biometrics, 1977, 33(1): 159-174. [32] Vaswani A, Shazeer N, Parmar N, et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 6000-6010. [33] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[EB/OL]. [2026-04-17]. https://arxiv.org/pdf/1707.06347v1. [34] The Autoware Foundation.Freespace planning algorithms: Hybrid A* and Python extension[EB/OL]. [2026-04-17]. https://autowarefoundation.github.io/autoware_universe/main/planning/autoware_freespace_planning_algorithms/. [35] Han Z C, Wu Y W, Li T, et al. An efficient spatial-temporal tra-jectory planner for autonomous vehicles in unstructured environments[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(2): 1797-1814.
2026, Vol. 39 
